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SUMMARY 


Problem 


The  Instructional  systems  development  (ISD)  procedures  are  used  for  training 
development  by  all  three  military  services.  Because  of  the  complexity  of  these 
procedures,  considerable  skill  is  required  in  applying  them.  Therefore,  to  aid  training 
developers,  the  Navy  Personnel  Reseat  ch  and  Development  Center  has  developed  a 
methodology  known  as  the  instructional  quality  inventory  (IQI),  which  is  designed  to 
"parallel  and  supplement"  ISD  procedures.  To  date,  evaluation  of  the  IQI  has  been 
indirect.  More  direct,  quantitative  evaluation  is  needed. 

Purpose 

The  objectives  of  this  research  v/ere  to  (1)  determine  whether  relatively 
inexperienced  persons  could  use  IQI  to  identify  difficulties  in  and  revise  instruction 
materials,  (2)  evaluate  the  effectiveness  of  the  revised  materials,  and  (3)  determine  costs 
of  applying  the  IQI. 

Approach 

The  optical  character  reader  (OCR)  module  from  the  Radioman  (RM)  "A"  School 
curriculum  was  selected  for  revision.  This  module  is  used  to  train  RM  students  to  verify 
information  contained  on  DD  Form  173,  which  is  optically  scanned  for  computer-based 
transmission.  The  computer  rejects  all  messages  that  do  not  meet  critical  requirements 
as  to  spelling,  punctuation,  and  character  alignment.  This  module  had  been  identified  by 
instructors  as  needing  revision. 

IQI  techniques  were  used  to  evaluate  the  then-current  material.  A  two-person  team 
with  limited  experience  in  the  use  of  the  IQI  process  analyzed  the  selected  material, 
examining  the  objectives,  testing  format  and  content,  and  training  materials.  This 
resulted  in  (1)  some  changes  to  the  objectives,  which  were  found  to  be  inconsistent  with 
job  requirements,  (2)  an  entirely  different  approach  to  testing,  and  (3)  a  new  set  of 
training  materials.  During  the  process,  lessonware  development  costs  were  tracked  to 
allow  for  later  cost  analysis. 

To  evaluate  the  effectiveness  of  the  revised  materials,  two  groups  of  30  students 
each  were  selected  at  random  from  the  RM  School.  One  group  (standard)  received  the 
existing  training  and  testing,  which  required  that  they  meet  test  criteria  at  several  stages 
of  the  training  before  they  could  advance.  The  second  group  (IQI)  received  the  revised 
training,  which  allowed  them  to  proceed  at  their  own  pace,  taking  self-tests  to  assess 
their  progress.  The  performance  of  the  two  groups  was  compared,  based  on  the  scores 
achieved  on  the  IQI  test. 

Findings  and  Discussion 

1.  The  IQI  provided  a  useful  framework  for  determining  deficiencies  in  current 
instruction.  By  examining  module  components  in  the  sequence  called  for  by  the  IQI, 
deficiencies  could  be  readily  identified  and  revisions  made. 

2.  While  applying  the  IQI,  several  problems  were  encountered.  The  most  prominent 
of  these  is  that  IQI  manuals  direct  the  user  to  relaxe  training  to  initial  job  skills,  but  they 
do  not  indicate  how  users  are  to  obtain  this  information.  Also,  because  of  the  many  small 
details  in  IQI  manuals,  the  user  can  be  distracted  from  seeing  the  overall  scope  of  the  task 
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(e.g.,  ensuring  a  correct  and  complete  set  of  objectives).  One  other  relatively  minor 
difficulty  occurred  in  distinguishing  between  use-aided  and  use-unaided  procedures. 

3.  The  groups  differed  significantly  on  their  test  performance.  IQI  students 
averaged  57  percent  correct  answers,  compared  to  49  percent  for  standard  students.  The 
standard  group  had  studied  their  materials  for  an  average  of  11.6  hours,  compared  to  2.8 
hours  for  the  IQI  group.  Thus,  the  IQI  group  scored  somewhat  higher  on  the  job-relevant 
test  than  trainees  in  the  standard  group. 

4.  An  important  consideration  in  evaluating  instruction  concerns  the  costs  incurred 
in  the  process.  These  were  almost  all  in  hourly  costs  for  the  research  team,  amounting  to 
1/5  of  a  man-year.  However,  it  is  estimated  that  this  figure  would  be  considerably 
reduced  if  IQI  were  applied  concurrently  with  the  ISD  process  and  would  be  less  as  the 
team  gained  experience. 

Conclusions 


The  findings  suggest  that  the  IQI  material  is  more  effective  in  producing  job-relevant 
performance  than  the  existing  materials.  How  long  it  would  take  for  trainees  to  learn  to 
a  high  test  performance  criterion  is  not  known.  The  fact  that  IQI  trainees  achieved  as 
well  as  they  did  in  2  to  3  hours  suggests  that  training  time  might  be  considerably  less  than 
that  required  currently  for  the  existing  materials. 

Recommendations 


1.  IQI  procedures  should  be  used  to  increase  the  quality  of  instructional  materials 
developed  by  ISD  users  and  others. 

2.  IQI  procedures  should  be  revised  to  provide  the  user  with  (a)  better  tools  to  deal 
with  the  job  relevancy  issue,  and  (b)  computer-based  direction  to  facilitate  completion  of 
the  many  steps. 

3.  Future  evaluations  should  be  made  in  situations  where  learning  conditions  are 
controlled  so  that  cost-effectiveness  can  be  calculated. 
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INTRODUCTION 


Problem 


All  military  instruction  is  presently  developed  in  accordance  with  the  interservice 
procedures  for  instructional  systems  development  (ISD),  which  were  conceived  as  an 
application  of  a  "systems  approach"  for  military  and  civilian  training  developers  (CNET, 
1975).  ISD  is  a  complex  process,  consisting  of  many  sequential  steps,  requiring  training 
developers  to  make  many  judgments  as  the  process  is  applied.  Because  of  the  large 
number  of  judgments  and  the  ambiguities  in  the  procedures,  the  final  instructional  product 
can  vary  widely  in  instructional  effectiveness  and  consistency  with  job  requirements.  To 
reduce  this  variability  and  ensure  appropriate  application  of  ISD  procedures,  the  instruc¬ 
tional  quality  inventory  (IQI)  (Wulfeck,  Ellis,  Richards,  Woods,  <5c  Merrill,  1978)  was 
designed.  Although  the  IQI  incorporates  prescriptions  derived  from  empirical  research, 
little  direct  evidence  exists  as  to  hv.w  well  people  can  use  the  procedures  and  the 
effectiveness  of  IQI  revised  instruction. 

Purpose 


The  objectives  of  this  research  were  to  (1)  determine  whether  relatively  inexperi¬ 
enced  persons  could  use  IQI  to  identify  difficulties  in  and  revise  instruction  materials,  (2) 
evaluate  the  effectiveness  of  the  revised  materials,  and  (3)  determine  costs  of  applying 
the  IQI  to  provide  a  basis  for  planning. 

Background 


IQI  is  basically  a  logical  method  that  uses  sound  principles  of  training  and  the 
psychology  of  learning  and  instruction  to  evaluate  the  quality  of  the  three  main  products 
of  instructional  development — objectives,  test  items,  and  instructional  presentation.  It 
deals  principally  with  cognitive  and  psychomotor  instruction  rather  than  attitudes  and 
motivation.  It  is  intended  for  use  by  people  who  are  familiar  with  ISD,  not  by  untrained 
personnel,  and  serves  as  a  supplement  to  ISD.  Since  IQI  assumes  that  what  needs  to  be 
taught  has  been  determined,  its  application  depends  upon  a  good  task  analysis  or  the 
availability  of  subject  matter  experts  (SMEs),  preferably  both. 

The  IQI  procedures  are  documented  in  four  reports:  an  introduction  and  overview 
(Wulfeck  et  al.,  1978),  a  user's,  manual  (Ellis,  Wulfeck,  &  Fredericks,  1979),  a  training 
workbook  (Fredericks,  1980),  and  a  job  performance  aid  (Ellis  and  Wulfeck,  1978).  IQI 
procedures  Include  the  following: 

1.  Since  all  ISD  steps  depend  on  careful  specification  of  learning  objectives,  the 
first  IQI  procedure  is  to  assure  the  adequacy  of  objectives.  This  is  done  by  classifying 
each  objective  and  judging  whether  or  not  it  accurately  reflects  the  intended  student 
performance  after  training.  Classification  is  determined  according  to  what  the 
student  is  required  to  do  with  the  information  he  learns  (remember  or  use)  and  what 
type  of  information  the  student  is  learning.  Objectives  are  considered  adequate  if  they 
meet  the  following  criterion: 

a.  They  must  have  properly  specified  conditions,  standards,  and  action.  Condi¬ 
tions  are  the  "givens"  in  a  particular  situation,  such  as  a  formula,  a  manual,  job  aids, 
tools,  etc.;  standards  refer  to  performance  levels  and  end  products;  action  specifies  the 
one  type  of  activity  sought  in  the  objective  (no  more  than  one  action  verb  per  objective  is 
permitted). 
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b.  They  must  be  uniquely  classified  on  the  task-content  matrix.  IQI  requires 
that  objectives  fall  into  only  one  cell  of  a  specially  developed  matrix  that  specifies  the 
content  type  of  the  objective  in  terms  of  (1)  requiring  trainees  to  learn  facts,  categorize 
things,  and  use  procedures,  rules,  or  principles,  and  (2)  what  the  task  level  is  (i.e.,  whether 
the  student  is  simply  to  remember  the  material  or  use  it  in  some  way). 

c.  They  must  be  consistent  with  the  job.  In  the  past,  classroom  objectives 
were  often  made  appropriate  to  the  constraints  of  the  teaching  environment;  their 
relationship  to  the  job  was  often  ignored  or  downplayed. 

2.  The  next  step  is  to  ensure  that  tests  accurately  measure  a  student's  progress 
toward  objectives.  This  is  done  by  assessing  the  consistency  between  test  items  and  their 
associated  objectives.  Essentially,  each  test  item  must  be  classified  in  the  same  way  as 
its  objective.  After  test  items  and  objectives  are  consistent,  the  adequacy  of  the  test 
items  is  assessed. 

3.  The  third  step  is  to  ensure  that  the  instructional  presentation(s)  is  (a)  consistent 
with  the  objectives  and  tests,  and  (b)  adequate  according  to  psychological  principles  of 
learning  (i.e.,  whether  it  will  support  student  learning). 

The  IQI  procedures  were  refined  through  the  use  of  training  workshops,  where  user 
difficulties  were  identified  and  procedures  classified.  However,  little  direct  empirical 
evidence  regarding  IQI  usability  and  effectiveness  is  available. 


REVISION  OF  TRAINING  MATERIALS 
Training  Development  Personnel 

A  two-person  team  was  formed  from  among  researchers  at  NAVPERSRANDCEN  who 
were  familiar  with  ISD  but  not  expel  ienced  in  using  IQI.  Their  intent  was  to  determine 
whether  the  IQI  procedures  could  be  used  to  revise  a  segment  of  a  Navy  "A"  School  course 
that  had  proven  to  be  unsatisfactory. 

Training  Materials  Selected 

The  optical  character  reader  (OCR)  form  module  from  the  Radioman  (RM)  "A"  School 
curriculum  was  selected  for  revision.  This  module  is  used  to  train  RM  students  to  verify 
message?  already  typed  on  DD  Form  173, 1  which  is  optically  scanned  for  computer-based 
transmission.  The  computer  rejects  all  messages  that  do  not  meet  critical  requirements 
as  to  spelling,  punctuation,  and  character  alignment.  Figure  l2  illustrates  a  copy  of  the 
DD  Form  173  currently  in  use.  Although  the  form  has  been  revised  since  the  work 
described  in  this  paper  was  completed,  the  requirements  remain  essentially  the  same. 


1  On-the-job,  RM  graduates  are  required  to  type  the  message.  However,  because  of 
the  lack  of  equipment  at  the  school,  the  ability  to  type  the  forms  correctly  could  not  be 
used  as  a  classroom  objective. 

2 Because  of  the  large  number  of  figures  in  this  section  relative  to  the  amount  of 
text,  the  figures  have  been  placed  at  the  end  of  the  section,  commencing  on  page  5. 
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The  entire  RM  "A"  School  curriculum  had  been  recently  revised;  however,  students 
were  having  difficulty  with  the  OCR  form  module,  taking  an  average  of  about  11  hours’ 
study  time,  including  remediation,  to  reach  criterion  on  the  tests.  Also,  the  school  had 
received  reports  that  students  were  unable  to  do  this  task  properly  upon  assignment  in  the 
fleet. 

IQI  Analysis 

As  indicated  previously,  the  team  members  had  had  prior  training  development 
experience  but  had  not  used  the  IQI.  The  team  attended  a  two-day  IQI  workshop  prior  to 
the  start  of  the  project  and  had  access  to  IQI  documentation. 3  Whenever  a  question 
arose,  one  team  member  first  sought  help  from  the  other  and  then  from  an  SME.  On  a 
few  occasions,  it  was  necessary  to  cor  ,ult  IQI  developers  to  clarify  an  IQI  application. 

The  team  reviewed  the  instruction  and  consulted  with  SMEs  in  the  school  and  a 
communications  station  to  become  familiar  with  the  OCR  form.  Frequent  and  continuous 
consultation  with  SMEs  was  necessary  during  the  IQI  analysis  to  verify  the  correctness  of 
materials,  determine  their  relationship  to  the  job,  and  clarify  technical  questions. 

Objectives 

The  team  started  with  an  analysis  of  the  objectives.  This  turned  up  minor  but 
persistent  problems.  In  most  cases,  the  objectives  lacked  time  standards  and  included  two 
action  verbs  (only  one  action  verb  per  objective  is  permissable).  Also,  provisions  were  not 
made  to  allow  for  the  use  of  an  alignment  margin  template,  which  can  be  used  to  check 
tab  stop  locations.  These  templates  are  available  on  the  job  and  can  be  used  easily. 
Objectives  were  revised  to  correct  these  deficiencies. 

Test  Items 


The  test  item  format  is  multiple-choice,  since  the  Navy's  computer-managed 
instruction  (CMI)  system  used  to  run  the  individualized  RM  course  allows  only  multiple- 
choice  answer  grading.  Although  the  test  items  in  the  original  OCR  module  are  multiple 
choice,  they  are  neither  consistent  with  the  objectives  nor  appropriate  to  the  job.  As  can 
be  seen  in  the  examples  in  Figure  2,  they  are  designed  to  test  for  facts,  rather  than  for 
use-unaided  procedures. 


be  given  to 


For  the  test  item  to  be  consistent  with  the  objectives,  a  completed  OCR  form  should 
the  student  for  verification  as  to  proper  spacing,  character  placement, 
spelling,  abbreviations,  and  punctuation.  This  necessitated  devising  'ome  procedure  that 
would  allow  the  use  of  the  OCR  form  in  conjunction  with  the  multiple-choice  format. 
Thus,  in  consultation  with  personnel  from  the  Instructional  Program  Development  Center 
(IPDC),  a  new  testing  format  was  designed  that  is  as  consistent  with  job  performance  as 
possible,  given  the  multiple-choice  testing  constraint  of  the  CMI  system.  Each  test 
question  is  actually  a  completed  OCR  form  with  one  or  no  errors,  as  shown  in  Figure  3 
The  student  had  to  verify  each  section  using  an  overlay  to  check  spacing.  He  then  had  to 
indicate,  using  the  multiple-choice  format  included  on  each  form,  the  section  in  which  the 
error  occurred  (there  was  no  more  than  one  error  in  a  message)  or  that  there  was  no  error. 


3At  the  time  of  the  analysis,  one  team  member  was  working  on  an  IQI  workbook, 
developing  practice  examples  of  IQI  concepts  (Fredericks,  1980)  This  may  have  resulted 
in  a  higher  IQI  skill  level  than  for  someone  who  had  only  taken  the  IQI  course. 
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Instructional  Presentation 


In  the  original  module,  information  was  presented  in  narrative  form,  broken  into 
various  components  (see  Figure  4).  After  the  student  finished  one  component,  he  took  a 
within-iesson  test  and  moved  on  to  the  next.  He  never  had  any  practice  looking  at  OCR 
forms  and  finding  any  errors  in  them. 

The  revised  training  materials  were  designed  to  reflect  job  conditions  as  closely  as 
possible,  given  the  limitations  of  the  school  facilities.  Since  verifying  the  form  is  a  use- 
unaided  procedure,  the  task  was  divided  into  steps,  the  order  of  which  followed  the 
completion  of  the  form  from  top  to  bottom.  Each  section  of  the  form  (e.  g.,  address 
component,  message  text)  v/as  separated  and  identified,  and  parts  within  the  sections 
(e.g.,  "From"  line,  "To"  line)  were  separated.  As  shown  in  Figure  5,  the  parts  were  identified 
in  the  left-hand  margin  of  the  revised  training  materials.  The  most  important  verification 
check  was  located  in  the  middle  column  (only  one  action  for  each  check),  and  additional 
information  was  provided  in  a  column  on  the  right. 

For  each  section  of  the  form,  the  following  v/as  included: 

1.  Self-test  c,jestions  for  use  in  pointing  out  important  points  and  helping  the 
student  focus  his  studying  (Figure  6). 

2.  An  example  cf  a  completed  OCR  form,  with  the  errors  identified  (Figure  7).  A 
checklist  form  was  used,  with  the  answers  given  for  each  par.,  of  the  section  (Figure  8). 

3.  A  second  example  of  a  completed  OCR  form.  This  time,  the  student  had  to 
identify  the  errors  using  a  blank  checklist  form.  After  he  identified  any  errors,  he  could 
check  another  OCR  form  where  the  errors  were  identified  as  feedback. 

4.  A  third  example  of  a  completed  OCR  form.  This  time,  the  student  had  to 
identify  the  errors  with  no  checklist  or  aids,  just  as  he  would  do  on  the  job.  After  he 
identified  any  errors,  he  was  shown  the  form  with  the  eirors  correct/1'1  and  a  <  arrow 
pointing  out  where  they  were  wrong. 

5.  More  examples  and  practice  were  given  the  student  as  he  went  through  the 
lesson.  This  was  experience  just  like  he  would  be  getting  on  the  job  except  it  was  in 
graduated  steps- -from  using  a  completed  checklist  to  a  blank  checklist  to  no  aids  when 
verifying  the  completed  OCR  for.-  . 

The  revised  materials  reflect  job  orientation  rather  than  the  topic  orientation  of  the 
standard  lesson.  Too  many  important  points  were  embedded  within  details  in  the  standard 
lesson.  The  revision  tried  to  identify  the  important  points  for  the  student  and  isolate 
them  into  steps  for  verifying  the  OCR  form. 

Cost  Analysis 

Course  revision  can  be  an  expensive  process,  especially  when  recently  developed 
programs  must  be  revised.  To  document  the  cost  of  revision,  provision  was  made  to  track 
software  development  costs  and  examine  training  costs,  if  applicable.  These  costs  mainly 
involved  personnel.  It  was  considered  that  costs  of  other  categories,  such  as  facilities  and 
hardware,  remained  fixed  for  both  the  original  training  development  approach  and  the  IQI 
revision. 
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Figure  1.  Illustration  of  the  DD  173  message  form. 
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You  may  use  NTP  3  SUPP-1,  as  necessary. 

1.  According  to  NTP  3  SUPP-1,  the  originator  of  the  given  DD  Form  173: 

a.  Should  read:  CQMSYSTORE  KEFLAVIK  UK. 

b.  Is  correct;  a  correction  is  required. 

c.  Should  read:  "NAVY  COMMISSARY  STORE  KEFLAVIK  IC." 

d.  Is  correct  as  it  is  typed  on  the  message  form. 

2.  According  to  NTP  3  SUPP-1,  the  INFO  addressee  on  the  given  DD  Form  173: 

a.  Should  read:  NAVEXHGSERCEN  SAN  FRANCISCO  CA. 

b.  Is  correct  as  it  stands. 

c.  Is  missing  its  geographical  location;  it  should  be  added. 

d.  Should  be  listed  on  the  message  form  using  the  long  title. 

3.  The  OPSIG  "ZEN"  is  used  by  an  originator  to  assure  that: 

a.  The  computer  reads  (accepts)  the  addressee  that  follows. 

b.  A  message  is  delivered  on  a  "Red"  circuit. 

c.  A  message  will  be  delivered  by  another  means. 

d.  The  message  is  sent  as  a  book  message. 

4.  Which  statement  below  is  true  about  verifying  the  address  component  on  DD 
Form  173? 

a.  The  long  title  address  of  commands  is  used. 

b.  No  punctuation  is  used  in  a  plain  language  address. 

c.  "XMT"  is  used  to  ensure  all  members  of  a  CAD  receive  a  message. 

d.  The  "FROM"  address  must  fit  into  a  maximum  of  two  lines. 


Figure  2.  Examples  of  standard  test  items. 
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Instructions  The  following  error  statements  correspond  to  the 

normally  found L/  answer  numbers  on  your  answer  sheet  as  shown  here: 
prior  to  test  i  t  i  **  j 

S'  D  0  0  0  0  I  Error  in  Headerline  Blocks 
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0  0  0  0  D  2  Error  in  Address  Component 
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Figure  3.  Example  of  an  IQI  test  item  meeting  the  multiple-choice  criterion.  In  this 
case,  the  error  is  in  the  headerline  blocks.  The  student  must  locate  the 
section  containing  the  error  and  mark  the  corresponding  alternative  on  the 
answer  sheet.  (For  illustration  only,  the  mode  of  marking  the  answer  is 
overlaid  on  the  test  OCR  item). 
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Address  Component 


Now  that  we  have  talked  about  the  headerline  blocks  and  the  baseline 
blocks,  let's  to  on  to  the  address  component. 

The  short  title  address  of  any  command  or  activity  is  always  used 
here.  These  addresses  are  listed  in  the  Plain  Language  Address 
Directory  (PLAD),  NTP  3,  Supplement  1.  Computers  accept  only  the 
listed  addressees,  so  the  addresses  must  be  correct.  If  there  is  any 
doub,,  check  the  PLAD.  The  PLAD  lists  the  long  and  short  titles  of 
activities  and  commands  alphabetically,  in  this  way: 

Long  Titles  of  Activities  Short  Titles  of  Activities 

Chapter  2.  U.S.  Navy  Chapter  3.  U.S.  Navy 

Chapter  4.  Coast  Guard  Chapter  5.  Coast  Guard 

Chapter  6.  Marine  Corps  Chapter  7.  Marine  Corps 

Chapter  10.  U.S.  Navy  Ships 
Chapter  11.  U.S.  Coast  Guard  Ships 

In  Figure  8  shown  on  the  next  page  you  will  see  examples  of  long  and 
short  title  listings  from  PLAD.  Note  how  commands  are  listed  in 
both  chapters. 


Figure  4.  Example  of  original  training  materials. 
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Address  Component 

The  OCR  scans  the  address  components  to  route  the  message.  Addressees  are  stored  in 
its  memory,  so  the  scanner  must  find  the  same  characters  as  those  stored  in  memory  for 
proper  routing.  These  addresses  are  known  as  "Plain  Language  Addresses"  (PLA)  and  are 
found  in  the  Plain  Language  Address  Directory  (PLAD)  of  NTP-3  (C),  Supplement  1.  You 
must  remember  +he  following  information  about  each  line: 


VERIFICATION 

CHECKS 


CONSTRAINTS  AND 
ADDITIONAL  INFO 


General 

Information 

Check  for  extra 
punctuation. 

1. 

PLAs  have  no  punctuation. 

Check  for  correct  numbers. 

2. 

Numbers  must  be  spelled  out. 

Check  for  abbreviations. 

3. 

States  are  abbreviated. 

From  Line 

Verify  PLA  in  PLAD. 

1. 

Allowed  one  line  only. 

Verify  PLA  as  that 
of  drafter's. 

2. 

Cannot  extend  beyond  tab  75. 

To  Line 

Verify  PLA  in  PLAD. 

1. 

Addresses  may  be  continued 
on  two  more  lines. 

Check  indentation  for 
continued  addresses. 

2. 

Continued  addresses  must 
be  indented  five  spaces  to 
tab  31  so  the  scanner  knows 
they  are  part  of  the  same 
address. 

Info  Line 

Check  for  pro  sign  INFO. 

1. 

The  first  info  addressee 
must  be  preceded  by  the 
pro  sign  INFO  (no  punctuation). 

2. 

All  other  info  addresses 
follow  without  the  pro  sign. 

3. 

If  there  are  no  action 
addresses,  the  pro  sign 
is  typed  over  the  "TO." 

Figure  5.  Example  of  IQI-revised  training  materials. 
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Address  Component 

a.  Addresses  must  be  looked  up  in  the _ . 

T  F  Multiple  addresses  have  punctuation. 

b.  The  from  line  contains  the  address  of  the _ . 

How  many  lines  may  this  address  occupy? 

c.  How  many  addressees  may  appear  on  an  OCR  form? 

How  can  the  computer  tell  if  an  address  is  continued  on  another  line? 

d.  How  do  you  identify  an  INFO  address? 

e.  How  do  you  delete  part  of  a  collective  address? 

f.  How  do  you  identify  an  addressee  that  is  to  receive  a  message  by  means 
other  than  electrical? 

g.  Messages  sent  to  commercial  addresses  must  be  classified  as _ . 

T  F  Commercial  addresses  may  only  extend  for  one  line. 

Figure  6.  Self-test  questions  used  in  IQI-revised  training  materials. 
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OCR  message  #1.  Example  with  errors. 


DD  ^73 


Figure  7.  Example  of  OCR  form  with  annotated  errors. 
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Verifying  Address  Components 


Verifying  addresses  is  one  of  the  most  important  functions  you  will 
perform  as  a  novice  RM.  Check  the  previous  section  on  address 
components  if  you  do  not  remember  all  the  information  you  need  to 
verify.  Use  the  template  to  ensure  correct  alignment  and  spacing 
and  the  PLAD  to  ensure  correct  addresses.  After  you  become 
familiar  with  these  addresses,  you  will  be  able  to  recognize  them 
from  memory.  Use  the  same  OCR  form  to  verify  the  steps  given 
below. 


_2k_  1. 
J&L 

2. 

3- 

7%  4. 
OK  5. 

Aid-  6- 
2k-  7. 


Punctuation  is  correct. 

Numbers  are  spelled  out. 

Appropriate  abbreviations  are  used. 

Drafter's  PLA  is  correct. 

Addressees'  PLAs  are  correct. 

Info  pro  sign  correct. 

Info  addressees'  PLAs  are  correct. 

Collective  addressees'  PLAs  are  correct. 

XMT  pro  sign  correct. 

ZEN  pro  sign  correct. 

Alignment  and  spacing  correct.  /ku&Ae.  dfeue- 


Figure  8.  Completed  checklist  form  for  OCR  form  shown  in  Figure  7. 
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EVALUATION  OF  REVISED  MATERIALS 


Subjects 

To  assess  the  effectiveness  of  the  revised  materials,  two  experimental  groups  of  30 
students  each  were  selected  at  random  from  the  RM  "A"  School.  One  group  was  called 
the  IQI  group  and  the  other,  the  standard  group.  To  assess  potential  prior  differences  in 
the  groups,  their  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  test  scores  were 
compared.  Analysis  of  these  scores  (see  Table  1)  showed  they  were  similar  except  for  the 
AFQT  score. 


Table  1 

Student  Characteristics 


Mean  Scores 

Groups 

Numerical 
AFQT  Operations 

Attention 

to 

Detail 

Word  Arthmetic  Math 
Knowledge  Reasoning  Knowledge 

Mechanical 

Information 

Standard 

61.8 

53.1 

52.3 

55.4  53.1  52.1 

f,n  / 

*t7  •  O 

IQI 

55.6 

54.0 

53.1 

54.3  51.6  51.7 

49.5 

Learning  Conditions 

The  standard  group  was  trained  to  verify  messages  on  DD  Form  173  using  the  original 
OCR  module.  Their  training  procedures  are  listed  in  Table  2.  They  had  to  meet 
performance  criteria  (on  test  items  similar  to  those  shown  in  Figure  2)  before  they  could 
advance.  All  training  was  individualized,  with  students  working  in  large  study  areas  under 
the  supervision  of  several  learning  supervisors.  Within  1  or  2  days  after  they  completed 
the  OCR  module  successfully,  they  were  administered  an  11 -item  IQI  test.  Each  item 
consisted  of  a  completed  OCR  form,  which  was  either  correct  or  contained  one  or  no 
error  (similar  to  Figure  3).  Students  had  to  complete  the  mjltiple-choice  test  format  for 
each  item. 

The  IQI  group  was  trained  using  IQI  revised  materials  approximately  2  months  after 
the  standard  group.  These  students  were  selected  randomly  from  the  same  RM  school 
population  just  before  they  were  ready  to  start  the  OCR  module.  A  nonschool  chief  petty 
officer  (CPO)  explained  to  the  students  that  they  were  being  given  a  different  version  of 
the  OCR  module.  All  training  was  done  in  one  of  the  regular  study  areas  under  the  CPO's 
supervision.  Upon  completion,  the  IQI  group  was  administered  the  IQI  test  (see  Table  2). 
Since  the  RM  School  would  not  agree  to  substitute  the  revised  OCR  materials  for  the 
original  materials  as  far  as  class  completion  credit  was  concerned,  the  IQI  group  also  had 
to  take  the  regular  within-lesson  and  final  module  tests. 

The  IQI  test  was  used  to  assess  the  performance  of  the  two  groups  because  it  was 
designed  to  be  consistent  with  the  revised  objectives  and  with  the  job,  which  was  not  true 
of  the  standard  module  examinations.  Because  of  this  distinction,  performance  on  the  IQI 
test  provides  a  basis  for  comparing  the  groups  that  reflects  the  potential  for  job 
performance. 
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Table  2 


Training  Procedures  for  Instructional  Evaluation 


Group 

Instruction 

Study  Procedures 

Standard 

Standard  CNET  approved 
self-study  narrative  or 
programmed  instruction 
(PI)  with  mandatory 
within  lesson  tests. 

1. 

2. 

Student  studies  either  form  of  lesson 
until  ready  to  take  sub  lesson  test. 

If  student  fails,  repeat  either  nar¬ 
rative  or  PI  up  to  three  times  until 
test  is  passed. 

3. 

If  student  passes,  go  on  the  next  sub¬ 
lesson,  and  repeat  procedure. 

4. 

Upon  completion  of  the  sublessons,  the 
student  takes  a  self  test. 

5. 

When  satisfied  with  the  results,  the 
student  takes  the  final  full  lesson 
test. 

6. 

One  or  2  days  later,  the  student  takes 
the  IQI  test. 

1QI 

Experimental  self-study 
narrative  with  self 

1. 

Student  studies  and  takes  sublesson 
self  tests. 

tests. 

2. 

When  satisfied,  student  goes  on  to 
next  section,  and  repeats  procedures. 

3. 

Upon  completion,  takes  IQI  test. 

Group  Performance 

The  mean  percent  correct  for  each  group  on  the  IQI  test  is  shown  in  Table  3,  aiong 
with  the  average  number  of  hours  taken  to  study  the  OCR  module.  As  indicated  earlier, 
the  two  groups  differed  on  the  AFQT  score.'  To  correct  for  this  difference  and  to 
increase  the  precision  of  the  test  score  measures,  a  one-way  analysis  of  covariance  was 
performed  on  the  data.  When  corrected  for  differences  in  AFQT  scores,  the  IQI  student 
test  scores  were  significantly  higher  than  the  standard  group. 

The  obvious  and  large  difference  in  mean  study  time  shown  in  Table  3  for  the  two 
groups  is  misleading.  The  groups  were  studying  under  very  different  requirements.  As 
noted  earlier,  the  standard  group  was  required  by  the  school  to  learn  materials  and  take 
intermodule  tests  that  were  not  closely  job-related.  The  IQI  group  was  measured  solely  on 
the  time  taken  to  complete  the  IQI  materials.  The,'1  conditions  probably  produced  most 

'The  difference  in  AFQT  scores  may  be  a  function  of  the  lapse  in  time  between  group 
testing— students  entering  the  Navy  at  different  times  of  the  year  tend  to  have  different 
aptitude  levels  (BUPERS,  1980). 
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of  the  difference  in  study  time.  It  is  not  unusual  to  be  unable  to  run  approximately 
matched  groups  in  naturalistic  settings,  but  It  does  limit  the  generality  of  the  results  and 
weakens  the  conclusions.  However,  the  data  in  Table  3  suggest  that  the  materials 
produced  under  .QI  guidelines  produce  the  kind  of  performance  the  fleet  requires, 
probably  in  less  time  than  the  present  training.  This  should  be  tested. 


Table  3 

Student  Performance  (Unadjusted  scores) 
and  Results  of  Covariate  Analysis 


Standard 

IQI 

F 

IQI  Test 

Score  (%  correct)  49 . 1 

57.0 

6.07* 

Mean  Study  Time 

(hours)  11. 6a 

2.8 

Standard  Deviation  5.2 

1.0 

includes  restudy  time  to  reach  RM  school  test  criterion. 
*p  <  .05. 


DISCUSSION 


IQI  Analyses 

Usability 

The  IQI  provided  a  useful  framework  for  critical  examination  of  all  elements  of 
instructional  design.  If  used  in  the  manner  intended,  it  takes  the  instructional  designer 
step  by  step  through  all  critical  elements,  which  are  no i  fully  specified  in  the  ISD  process. 
The  crux  of  the  IQI  analysis  is  ensuring  that  training  and  testing  are  relevant  to  the  job. 
By  forcing  the  designer  to  focus  on  job  relevant  behaviors,  extraneous  training  and 
inappropriate  testing  are  minimized. 

In  the  present  study,  IQI  analysis  revealed  areas  where  the  current  instruction  did  not 
provide  adequate  training  or  was  otherwise  faulty.  For  example,  many  objectives  did  not 
specify  performance  standards;  thus,  the  degree  of  precision  or  the  amount  of  time 
necessary  for  a  given  task  had  to  be  inferred.  By  following  the  stepwise  IQI  procedures, 
these  deficiencies  were  readily  identified.  If  the  purpose  of  using  the  IQI  is  course 
revision,  as  in  this  case,  redesign  work  can  readily  be  directed  to  specific  problem  areas. 
On  the  other  hand,  if  the  purpose  is  course  development,  IQI  techniques  will  aid  in 
ensuring  proper  initial  development  and  minimize  further  revision,  independent  of  the 
particular  training  development  method. 

Problems  Encountered 


While  the  IQI  manuals  provide  detailed  guidance,  some  difficulties  were  encountered, 
particularly  in  determining  the  job  relevancy  of  the  objectives,  the  training  materials,  and 
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the  testing  designed  to  measure  performance  on  the  objectives.  The  IQI  manuals  direct 
the  user  to  "remember  the  job"  or  to  determine  whether  the  objectives,  etc.,  are 
appropriate  to  the  initial  work  conditions;  however,  it  is  not  clear  where  users  are  to 
obtain  information  about  the  job.  If  it  is  implied  that  course  designers  will  have  access  to 
SME,  this  needs  to  be  clarified.  Also,  other  potential  sources  of  job  performance  criteria 
need  to  be  specified  because  of  the  difficulty  in  obtaining  SMEs.  Since  training  carries 
the  connotation  of  ultimate  job  performance,  the  source(s)  of  this  information  needs  to  be 
fully  explored. 

Another  difficulty  may  be  inherent  in  IQI  analysis;  that  is,  because  of  the  require¬ 
ment  for  many  small  details,  the  designer  may  be  distracted  from  the  overall  view  of 
what  the  instruction  is  supposed  to  convey.  Building  the  information  bank  from  which  the 
ultimate  instruction  is  constructed  can  be  a  tedious  process.  For  example,  determining 
objective  adequacy  requires  as  many  as  26  separate  judgments.  When  there  are  a  large 
number  of  objectives,  this  task  can  be  overwhelming.  It  would  be  desirable  to  have  the 
designer  stand  back  from  the  stepwise  process  at  regular  intervals  to  integrate  the  many 
judgments  with  the  overall  process.  When  working  with  objectives,  it  is  still  necessary  to 
ensure  that  they  include  all  job  relevant  tasks. 

An  additional  problem  involved  distinguishing  use-aided  tasks  from  unaided  tasks. 
The  IQI  defined  an  aid  as  "anything  that  replaces  the  need  for  memory  ...  ."  However, 
this  distinction  was  not  sufficient  to  allow  one  team  member  to  discriminate  between  aids 
and  what  the  IQI  terms  "»'-ols."  For  example,  while  the  IQI  manuals  classify  a  list  or  a 
table  as  an  aid,  they  classify  the  template  used  by  an  RM  to  verify  spacing,  etc.,  as  a  tool. 
While  this  may  be  a  minor  point,  an  educational  specialist  making  this  misclassification 
would  then  assign  the  wrong  task  level.  This,  in  turn,  would  cause  problems  in 
determining  the  appropriateness  of  the  objective  and  the  consistency  of  the  test  items  and 
instructional  presentation. 

Cost  Considerations 


The  only  nontrivial  cost  accruing  to  this  application  of  IQI  procedures  was  for  lesson 
development;  there  was  no  need  for  additional  hardware.  Each  team  member  took 
approximately  200  hours  over  a  5-month  period  to  evaluate  and  revise  objectives,  evaluate 
and  develop  test  materials,  and  revise  course  materials.  The  cost  for  this  team  was 
approximately  1/5  of  a  man-ycar.  However,  if  IQI  techniques  were  applied  to  instruction 
when  it  is  initially  developed  by  an  experienced  team,  this  cost  should  drop  significantly. 

Alternate  Version  Analysis 


Since  Instructional  Program  Development  Center  (IPDC)  personnel  considered  the  IQI 
revision  as  experimental,  they  initiated  a  parallel  revision  effort  as  part  of  an  effort  to 
modify  the  majority  of  the  RM  curriculum.  As  the  two  revision  teams  were  using  similar 
principles  to  modify  the  OCR  module,  a  brief  review  of  similarities  and  contrasts  in 
execution  is  provided. 

1.  Objectives.  The  RM  school  has  an  equipment  limitation  that  prevents  training 
students  in  actually  preparing  (typing)  OCR  forms.  However,  the  IQI  team  did  develop 
objectives  for  form  preparation  requirements  that  called  for  memorization  only,  such  as 
setting  the  typewriter  tab  stops  and  recalling  constraints  on  constructing  address 
components.  The  IQI  objectives  also  included  more  complete  standards  for  student 
performance,  including  both  time  to  complete  the  task  and  required  accuracy  of  the 
finished  product.  In  other  respects,  the  IQI  and  IPD  sets  of  objectives  were  similar. 
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2.  Test  procedures. 


a.  The  IPD  and  IQI  end-of-modole  tests  are  identical,  since  they  were 
developed  as  a  joint  effort.  As  indicated  previously,  test  items  consist  of  a  completed 
OCR  form;  the  student  must  select  the  portion  of  the  form  that  contains  an  error. 

b.  The  IPD  intramodule  tests,  which  are  used  as  progress  checks,  are  similar  in 
format  to  the  final  module  exam.  However,  testing  is  restricted  to  the  particular  lesson 
topic,  which  limits  the  number  of  alternatives  on  the  form  that  can  contain  an  error.  In 
the  IQI  version,  the  student  did  not  have  to  pass  intramodule  tests.  Progress  is  checked  by 
student  self-tests. 

3.  Instructional  presentation.  Overall,  the  differences  between  the  two  versions 
are  minor,  although  they  involve  slightly  different  levels  of  emphasis.  In  both,  the 
objectives  call  for  remembering  facts  ard  performing  use-unaided  procedures.  However, 
the  IPD  version  does  not  provide  practice  in  remembering  facts  and,  until  the  last  section, 
does  not  provide  practice  in  verifying  steps.  The  IQI  version  instruction  provides  practice 
in  both,  and  gradually  reduces  the  prompts  used  to  familiarize  the  student  with  the 
necessary  steps.  Specific  differences  that  were  found  include: 

a.  The  IPD  version  includes  information  not  found  in  the  IQI  version,  including 
a  description  of  the  difference  between  10  and  12  pitch  lettering,  multiple  page 
header  lines,  a  mere  complete  description  of  template  use,  and  information  on  readdressals 
without  PSN  numbers  (some  of  this  information  was  not  available  to  the  IQI  team).  Also, 
the  IPD  version  provided  better  and  more  complete  illustrations  throughout.  Because  the 
IQI  version  was  experimental  in  nature,  it  was  not  illustrated  extensively  in  an  attempt  to 
reduce  production  time  and  costs. 

b.  The  IQI  version  provides  information  missing  in  the  standard  version:  better 
separation  and  identification  of  areas  that  must  be  correct  before  transmission  can  occur, 
information  on  the  ZEN  prosign,  and  nontest  practice  on  checking  OCR  forms  and 
readdressals. 


CONCLUSIONS 

The  application  of  IQI  facilitates  the  use  of  ISD  procedures  in  ensuring  job  relevant 
instruction.  It  is  especially  useful  in  locating  and  preventing  omissions  by  directing  the 
user  to  examine  all  steps  in  instructional  development.  In  this  instance,  instructional 
problems  and  errors  were  readily  apparent. 

Results  of  student  performance  on  the  IQI  test  indicate  that  higher  levels  of 
performance  are  possible  in  skills  that  are  more  directly  applicable  to  fleet  needs. 
Although  better  controls  are  needed  to  conclude  whether  acceptable  fleet  performance 
levels  can  be  obtained  efficiently,  the  evidence  of  this  study  indicates  these  standards  can 
be  met. 


RECOMMENDATIONS 

1.  Since  the  IQI  procedures  provide  a  useful  aid  in  developing  instruction  and  in 
enhancing  job-relevant  training,  they  should  be  incorporated  in  Navy  training  develop¬ 
ment. 
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2.  IQI  procedures  should  be  modified  to  provide  the  user  with  (a)  better  tools  to 
deal  with  the  job  relevancy  issue  and  (b)  computer-based  direction  to  aid  in  the  many 
decisions  that  must  be  made  in  designing  instruction. 

3.  Future  evaluations  of  IQI  applications  should  be  conducted  with  control  over 
learning  and  testing  conditions  for  all  groups  so  that  information  is  provided  for  decision 
making. 
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